A safe data analysis environment for astronomical data centres

The volume of data available to astronomers via the Internet is doubling every twelve months or so. This is due both to the increase in the number of data sources (astronomy is an active field, while archives of old observations retain scientific value) and in the size of individual data sources (multi-TB datasets are the norm in the current generation of sky surveys). The new sky surveys are generally motivated scientifically by large-scale statistical analyses, such as searches for rare objects (e.g. high-redshift quasars or brown dwarfs), measurements of the clustering strength of galaxies or the production of summaries of the properties of a particular source population. In many cases, these analyses may require the integration of data from several sources (which is the motivation for the creation of the Virtual Observatory1), but, even when an analysis uses data from only one source, it will often require more data than an astronomer can readily download to a local workstation. This motivates the consideration of methods to allow astronomers to upload analysis code to be run at the data centre, rather than having them download the data. This is clearly fraught with potential security problems, and requires the construction of a safe environment within which users can run their analysis code, without risking the integrity of the data centre’s holdings. The Research Agenda section of the successful proposal by the CANDO consortium to host the Digital Curation Centre (DCC) presented the design and prototyping (via an astronomical testbed) of just such a safe data analysis environment as one of its priorities, and the goal of this document is to initiate that work. In what follows we discuss some of the requirements that astronomers have of such an environment, as well as outlining some possible ways of meeting them, some of which require no more than the implementation of well-known techniques, while others lie at the boundary of what is current research in computer science. Our aim here is to define more closely the scope of, and plans for, the research on this topic envisaged in the DCC proposal.